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Abstract 

A human-centered approach to computer systems design 
involves reframing analysis in terms of people interacting 
with each other, not only human-machine interaction. The 
primary concern is not how people can interact with 
computers, but how shall we design computers to help 
people work together? An analysis of astronaut interactions 
with CapCom on Earth during one traverse of Apollo 17 
shows what kind of information was conveyed and what 
might be automated today. A variety of agent and robotic 
technologies are proposed that deal with recurrent problems 
in communication and coordination during the analyzed 
traverse. 


DATA AND ANALYSIS SUMMARY 

During the 75 hours on the Moon, the Apollo 17 crew 
conducted three EVAs totaling 22 hours on the lunar 
surface. These EVAs included lunar rover traverses 
totaling 36 km, collection of lunar samples at 22 
locations in the Taurus-Littrow Valley, deployment or 
performance of 10 science experiments, and 
examination and photography of the lunar surface. 

(Jones, 1999) 

The present analysis is motivated by a problem and an 
opportunity. The problem is that the accomplishments of 
the Apollo 1 7 crew during their three days on the moon 
could not be sustained for weeks at a time, let alone months 
on Mars. The pace was too fast; the amount of work and 
difficulties in navigating and using tools too strenuous. 
Furthermore, the assistance provided by CapCom that 
made the work efficient at all will be unavailable from 
Earth during" Mars Efface" operations due to the lG-40 
minute roundtrip time delay. 

The opportunity arises from the observation that 
CapCom provided many services that could be easily 
automated: 

146:30:19 Ceman: Say, Bob, where can I get a new 
set of bags? 

146:30:23 Parker: Okay, you want new bags... They 'll 
be under Jack's seat. 

Automating the CapCom role would not only make a Mars 
mission possible, but also make the work less burdensome 
than it was on the moon, because many of the verbal 
interactions with the crew would be replaced by telemetry. 


Furthermore, CapCom provides a model of a 
disembodied agent (not a robot), whose coordination role is 
distinctly different from the surface crew. He does not 
work on the same tasks; he does not carry out any physical 
work. Past research emphasis on computers as robots has 
almost totally missed the opportunity to develop software 
agents (Jennings, et al., 1998; Alonso, 2002) for assisting 
in surface exploration. Analysis shows that an agent need 
not be a collaborator, but an assistant — a remote 
agent — that logs, tracks, advises, and monitors the work. 
Obviously, the functions of a physical robot and software 
agents could be combined. But first we need to understand 
what services people provide when playing this assistant 
role from a distance. 

To reveal the function of CapCom, I categorized and 
analyzed examples from Apollo 17 of the interaction 
between the surface EVA crew (Schmitt and Ceman) and 
CapCom on Earth (Bob Parker). This analysis is based on 
three segments of the second EVA during Apollo 17, 
termed “Orange Soil,” “Traverse to Station 5, ’’and 
“Geology Station 5 at Camelot Crater” in the Apollo Lunar 
Surface Journal (ALSJ; Jones, 1999). Mission elapsed time 
is continuous from 145:23:48 to 146:56:34 (about 1.5 hrs). 

Analysis of the transcripts reveals the following broad 
categories of information flow and work management 
functions as the EVA crew interacts with CapCom 
throughout their work: 

1 . Reading out information (logging): 

a. sample bag numbers 

b. photography frame counts 

c. rover systems indicators 

2. Asking where materials (cans, bags) are located 

3. Providing descriptions' (geological, equipment 
condition) for the record 

4. Suggesting, requesting, or documenting 
equipment settings and usage (e.g., suit cooling, 
film magazine change, dusting radiators) 

CapCom actively manages the work on the lunar surface: 

1. Indicates elapsed time, time remaining at a site, 
including walkback (turn around) warnings 

2. States revised plans for substituting, skipping, or 
reprioritizing work 

3. Provides navigation advice, including identifying 
craters the crew is seeing 


‘ Also, Institute for Human and Machine Cognition, UWF, Pensacola, FL. 


These interactions occur in ordinary conversations, with 
many complications involving disruptive, misheard, and 
mistaken remarks. These interactions and problems are 
illustrated further after a discussion of broad technological 
approaches for dealing with the problems. 

PROPOSED SURFACE EVA 
TECHNOLOGIES 

Analysis of the transcript suggests four technologies for 
dramatically improving surface exploration efficiency: 
Telemetry between rover and mission support, e.g., 
rover battery temperature, film frame numbers, bag 
numbers. Many verbal interactions between the crew and 
CapCom involve reading information that could be 
automatically provided by telemetry. Bag numbers might 
be transmitted by a scanning device on the rover or the suit 
sleeve. 

A ^‘remote agent,” resembling Bob, with whom the 
crew may have a mixed-initiative dialog about their work, 
while coordinating this work with a remote crew (at the 
Hab or Libration Point) and with Earth. During Apollo 17, 
Bob plays a distinct role, not like a third member of the 
EVA crew, but as a remote advisor for recording 
information, helping find and use equipment, and 
especially to prioritize and time activities. Although it may 
be tempting to refer to Bob as a member of the EVA team, 
the crew relates to him very differently from the way they 
interact with each other. In many respects. Bob is more 
present to each of them individually than they are to each 
other. That is, they individually coordinate their actions 
and narrations more closely with Bob than they do with 
each other. 

A navigation-task display with a map overlaid, 
indicating the current location, the route so far, and the 
planned route with stops. Timings indicate how much 
time was spent at previous stops, projected stop times 
(adjusted for current performance and schedule changes), 
and time remaining. The display indicates the current 
activity and time allocation, plus the next activity. 

Bag holder and logger-the astronauts spend a lot of 
time holding a bag open while the other shovels in soil or 
inserts a rock, or simply carrying sample bags: 

146:43:02 Schmitt: Yep. Whew. I've got to have Gene 
with me since (I) can't ca^ sarnple bags. 
i46:43:09 Parker: Roger. 

146:43:10 Schmitt: I probably can if I'm careful; but I 
keep dropping them.. 

Having a robot assistant would free up the other crew 
member. The robot could also automatically scan the bag 
number, so it needn’t be read out loud by the astronaut. 
This is a classic example of how examining work practice 
reveals needs that “technology push” would not consider. 
Assisting the crew in routine tasks should be considered 
before “autonomous robots” or doing infrequent big jobs, 
such as unrolling cables. 

The most demanding aspects of this proposed system are 
understanding natural dialogue, especially interleaved 


conversations — Crew 1 & Crew 2; CapCom & Crew 1; 
CapCom & Crew 2 — and tracking what the crew is doing. 

CREW-CAPCOM INTERACTIONS IN 
APOLLO 17 EVA 2 

Excerpts are sequential (mission elapsed time) within each 
category. Comments in brackets [...] are in the original 
transcript. 

Crew: Logging sample bag numbers 

Bag information includes source of sample and description 
145:33:49 Ceman: Bag 509 has got the orange material 
from, oh, about 2 to 3 inches down. 

Bob usually responds '‘Copy that’* 

145:34:53 Ceman: Bob, the gray material that's adjacent to 
the red material is in - what would I say - (bag) 510. 
145:35:01 Parker: Copy that. 

Descriptions may be elaborated for a turn or two and 
collaborative: 

145:35:37 Schmitt: 5 1 1 has the gray from the other side of 
the orange band. 

145:35:41 Ceman: And the other side happens to be the 
crater side. 

145:35:46 Schmitt: That's right. North side. 

Bob has to be listening for information directed at him 
145:46:10 Schmitt: (To Gene) Okay, I got it. (To Houston) 
Okay, the basalt (from the large boulder) is in bag 512. 

Bob confirms that he has heard correctly 
145:48:21 Parker: Okay, and. Jack, I copied - aside from 
three trench samples - 1 copied one single bag of basalt 
samples. Is that correct? 

145:48:35 Schmitt: That's right. 512. 

145:48:36 Parker: Copy that. 

146:09:30 Ceman: Bag 43 Yankee. 

146:09:33 Parker: Copy; 43 Yankee. 

Schmitt has been scooping soil; Ceman tells Bob which 
bag was used 

146:44:56 Ceman: 455 is that bag number, Bob. 

.146:44:57 Parker: Copy that. 

The crew often works together to take and secure a sample, 
so descriptions may come from either of them. Here Gene 
describes Jack's sample (465). Bob anticipates the bag 
number for the rock sample (466). Then Jack changes the 
topic back to the previous sample (465) to add information. 
146:45:37 Schmitt: (Pouring) I think we better leave it at 
that. 

146:45:43 Ceman: Okay, 465. Pick that other one up and 
I'll bag it real quick. 

146:45:45 Parker: Copy that. 
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146:45:49 Ceman: That’s the soil from on top the rock. 
And we’re taking a piece of the rock itself, which looks 
pretty much like the other one, Bob. It might be a little bit 
more vesicular. 

146:46:00 Parker: Okay, and that'll be in 466, right? 
146:46:06 Ceman: You’re right again. Here we are and I’ll 
be able to grab it with my hand. If I put this away. (Pause) 
Okay. (Pause) 

146:46:23 Schmitt; Okay, the soil came from a half a 
meter in from the soil boundary. ... 

Crew: Logging photography frame counts and planning 
usage 

Frame counts are provided for both the commander (CDR) 
and lunar module pilot (LMP) 

145:55:15 Schmitt: And, Bob, LMP is at (frame count) 
seven-five 

Bob requests a frame readout, telling Schmitt to keep using 
the same magazine. The readout process serves as an 
opportunity to note the need to change the magazine. If 
frame logging is automated, an agent woidd need to take 
on this function. 

146:53:16 Parker: And how about a frame callout before 
you get back on, guys. 

146:53:19 Ceman: Got it. 

146:53:23 Schmitt: Yeah, I need some new... (Responding 
to Bob) Do you want me to get it (a new magazine) here? 
(Pause) 

146:53:33 Ceman: CDR's at fifty. 

146:53:34 Parker: Copy that. 

146:53:35 Schmitt: A hundred and seventy. 

146:53:36 Parker: Copy one seven zero. 

146:53:37 Schmitt: LMFs 170. 

146:53:41 Parker: And, Jack, it'd be my opinion, since 
you’re just going back over the same path, that you came 
up this morning, it’s probably not necessary. 

146:53:49 Schmitt: Okay, I'll use it until it runs out. 
146:53:50 Parker: Okay. 

146:27:44 Schmitt: Bob, I have 135 frames. I think I can 
finish the station, don’t you? 

146:27:51 Parker: Yes, probably. (Long Pause) 

Crew: Logging rover systems indicators 
-145:53:39-Ceman: -(-The 'gravimeter-r-eading is) 67-0, 012, 
501; 670, 012,501. 

146:26:58 Schmitt: Okay. Oh, the (SEP) temperature; 
they’d like to know. 

[Jack goes to the SEP receiver behind his seatf 
146:27:06 Schmitt: Temperature is still about 1 12. 
146:27:08 Parker: Copy that. 

As Cernan indicates, it is tedious to be the ''voice’' of the 
instruments; notice that he is also interpreting the readings 
146:27:27 Parker: Okay, and, Gene, if you’re not off the 
Rover, how about the rest of the Rover readouts? 


146:27:33 Ceman: Okay, Bob, I'm off, but I'll get them for 
you. I’m sorry. I look at them, and they all look good to me. 
And, you know, I keep forgetting to give them to you. 

Crew: Asking where materials are located 
145:35:28 Ceman: If I can remember where we put it. Bob, 
where did we put the small can? 

145:35:30 Schmitt: It’s in bag 7 under my seat. 

Notice how Cernan relies on Bob for this information, even 
though Schmitt showed before that he knows where things 
are stored 

146:30:19 Ceman: Say, Bob, where can I get a new set of 
bags? 

146:30:23 Parker: Okay, you want new bags... They ’ll be 
under Jack's seat. 

146:30:26 Schmitt: Under my seat, there's some, Geno. 
146:30:30 Ceman: Okay. Just loose? 

Crew: Providing descriptions Tgeological & equipment 
condition) for the record 

146:20:40 Schmitt: Bob, the fragment population - we're at 
099/2.0 - is still about the one-percent category of.„And it's 
hard to tell, going into the Sun, what kind of blocks you're 
dealing with. But my guess is - well, more than a guess - 
(is that) most of them look like they’re slightly vesicular. 
And, in that regard, resemble the gabbros. 

146:21:19 Parker: Okay, copy that. 

146:31:47 Schmitt: . . . Bob, I have the impression that 
these blocks are buried up here (and) that the mantle does 
exist, even on Camelot. There are a few blocks that are 
lying out on the.. .(It) looks like they're lying more or less 
on the surface, but you can attribute those to craters that 
have dismpted the block field. 

146:32:24 Parker: Okay; good observation. Jack. 

The crew reports condition of equipment 
146:41:54 Ceman: You know I've worn the RTV (Room- 
Temperature Vulcanizing silicon rubber) off that hammer 
already. 

146:41:57 Schmitt: Yeah, I saw that. 

146:42:00 Parker: Roger, 17. Copy that. 

Crew. Bob: Suggesting, requesting, or documenting 
equipment settings and usage 

T45r34t00 ParkerrCopy that. (Long Pause) Okay, we’re- 
suggesting Intermediate (cooling) for you, Jack. 

145:26:04 Parker: And we're going to want the SEP 
opened and dusted as well here. With the switches turned 
off. 

146:01:15 Schmitt: Hey, Bob, I recommend that, if we 
ever do this again, let me get off and pick the charge off 
when we want to deploy it. It really adds to the fatigue of 
the hands. 

146:06:33 Schmitt: Aah. Stand by on pin 3, gang. 


146:06:35 Parker: Copying that. Remember to push it all 
the way back in, Jack, and start from scratch. 

146:00:11 Ceman: Hey, Bob, a note on those (battery) 
radiators: I have been dusting the covers at every stop, 
whether that’s any help or not. 

146:00:19 Parker: Okay; we copy that. (Pause) 

Bob: Indicates elapsed time, time remaining at a site, 
including walkback (turn around) warnings 
145:36:52 Parker: ...And the one problem at this station. 
Jack, is not that... 

145:37:01 Parker: ...we can decide priorities between this 
station or any other station. It's the fact that we're running 
up against the walkback constraints here in just a very few 
minutes, about two-zero (20) minutes. 

145:43:45 Parker: ...And, 17, for your thought... We have to 
be leaving here... Not "like". We have to be leaving here in 
fourteen minutes. On the move, because of walkback 
constraints. 

[Jack goes to the Rover. The walkback constraints are 
based on an assumed walking speed of 2. 7 km/hr. Houston 
thinks they are 4.2 kilometers from the SEP transmitter and 
4 kilometers from the LM, so the walkback would take an 
assumed 88 minutes. The EVA started at 140:35 and, in 14 
minutes, they will be 5 hours 22 minutes into it. An 88- 
minute walk back would get them back to the LM at 6 
hours 50 minutes. Although this estimate undoubtedly 
differs slightly from the calculation being made in 
Houston, the important point is that Houston has a firm 
requirement to get them back at a time close to 7 hours into 
the EVA f 

145:43:55 Parker: And we'd like to get a quick sample of 
the basalt up there on the rim, and Gene's stereo pan, and 
then press on. And I emphasize that it's walkback 
constraint we're up against in 14 minutes. 13 now. 

146:01 :02 Ceman: . . . (Pause) Bob, how long we been out? 
146:01:12 Parker: Stand by. 5 (hours) plus 26 (minutes); 5 
plus 26. 

146:25:46 Ceman: How's our time, Bob? 

146:25:48 Parker: Stand by. We're talking about that now. 
Stand by. You've got 25 minutes at this station, guys. 

We've given you somewhat of an extension here. You're 
using up some of it back atthe LM, but we've given you 
somewhat of an extension. You've got 25 minutes at this 
station. The primary priority will be subfloor documented 
samples, and then subfloor rake soil. (Pause) As you can 
imagine. 

146:26:26 Schmitt: Okay. 

Bob interrupts a conversation between the crew to remind 
them of the priorities, probably because of timing and lack 
of evidence that they are doing these tasks 
146:36:05 Parker: Okay, and a reminder, 17... 

146:36:07 Schmitt: Gene, if this is what you mean, it's... 


146:36:08 Parker: ...you guys, that the primary priority is 
the blocks and then a rake soil of the white subfloor soil 
there. And you’ve only got 15 minutes before we want you 
driving back to the LM. Over. 

146:36:22 Ceman: Okay. We'll get to work. Okay... 

Warnings are given at 25, 15, and 10 minutes — again 
interrupting the crew 's conversation; Bob must be able to 
hear '‘Yeah ” as a response directed at him. 

146:39:57 Parker: Okay, guys... 

146:39:58 Schmitt: I’ll get a... 

146:39:59 Parker: ...looks like you'll be going in about 10 
minutes. 

146:40:00 Schmitt: ...flight line photo. (Responding to 
Bob) Yeah. (Pause) (To Gene) \^y don't you get a flight 
line... 

Bob: States revised plans for substituting, skipping, or 
reprioritizing work 

145:24:47 Parker: Okay, and the number 1 and 2 priorities 
at this station will be samples from the crater rim and the 
pan from the crater rim. Over. 

145:36:14 Parker: 17, Houston. We'd like to get the double 
core here instead of the small can. Double core, please, 
instead of the small can. 

145:36:23 Ceman: Okay. 

145:36:24 Schmitt: Did you want it in the orange? 
145:36:26 Parker: Roger. That affirm. We can put cores in 
gray soil all the time. 

145:44:30 Parker: Why don't you leave the core there 
Gene, and you can take your stereo pan while Jack's getting 
that sample. And then you can get together and ram the 
core home. 

Bob confirms tasks the crew mentions to each other 
146:15:13 Ceman: We're coming up to 103 at 2.6 now, so 
we need a sample up here. 

146:15:17 Schmitt: Okay. 

146:15:18 Ceman: Okay. 103, 2.5. Anywhere. 

146:15:22 Parker: Roger. That's affirm. 

A task request is not a simple statement, but may involve a 
sequence of confirmations and elaboration — Bob appears 
to repeat the request, but notice that the first command he 
givesSchmitt gives the-objective (get theSEP ta-caol) but 
not how to do it (turn it off). Because of the apparent 
ambiguity, it requires FIVE statements (including Cernan 's 
repetition) before Schmitt is satisfied. 

146:26:28 Parker: As you get off, we'd also like to open 
the SEP and again get that to cool. 

146:26:36 Schmitt: Okay. You wanted to turn it Off? 
146:26:39 Parker: That's affirm... 

146:26:40 Ceman: Tiun it off 
146:26:41 Parker: ...turn it Off, open, dust... 

146:26:42 Schmitt: You want it off? 

146:26:43 Parker: ...the same thing we've been doing to it 
all aft(emoon). ..(correcting himself) all evening. 
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146:26:48 Schmitt: Well, it's midday here, Bob. (Pause) 

Bob replaces one task for another because of time 
pressure; this requires confirmation 
146:46:41 Parker: Roger. And the present time, we drop 
the rake soil, we’d just like to get the kilogram of soil 
somewhere between the boulders • (in) as open (a location) 
as you can, 

146:47:00 Schmitt: (Responding to Bob) Oh, you want a 
kilogram? 

146:47:02 Parker: Roger. 

146:47:03 Schmitt: From between the boulders? 

146:47:04 Parker: Roger. That’ll replace the rake soil 
sample we were going to get. And we'd like you moving in 
3 minutes. 

146:47:12 Ceman: (Joining Jack and responding to Bob) 
Okay. 

Crew. Bob: Provides navigation advice, including 
identifying craters they are seeing 
145:57:05 Ceman: Okay, I'm Min. Man, I'll tell you, that 
heading is going to put us right.. .Okay, Bob, give me 
a.. .Dang. Wait a minute. 

145:57:15 Parker: The heading you should be generally 
taking... 

145:57:17 Schmitt: Where we at? 

145:57:18 Parker: ...toward Victory is 090, Gene. 
145:57:23 Ceman: Okay, can you give me a bearing and 
range at Victory? 

145:57:27 Parker: Okay. Stand by. 

145:57:38 Parker: ...105 and 3.1. 

90 seconds later Cernan requests the information again 
145:59:03 Ceman: Hey, Bob, I forgot your numbers at 
Victory. How about giving them to me again? 

145:59:05 Parker: Okay. 105, 3.1. And it'll be a heading of 
090... 

145:59:11 Ceman: Okay. Thank you. 

145:59:12 Parker: ...that's the general heading in that 
direction, 

145:59:15 Ceman: Okay. 105, 3.1. 

Bob is tracking where they are and what they can see 
146:00:27-Schmitt: Okay, -sports fans. We're still about on 
the.. .Well, I think we moved.. .Yeah, we moved out into the 
Tortilla Flat area, I guess. (It's) not very flat. 

146:00:43 Parker: That's affirmative. (Pause) 

146:00:50 Ceman: Those kind (of shallow craters) I can go 
through...If I can see them coming. 

146:00:54 Schmitt: 102, 3.8. And where's Victory? 
146:01:01 Parker: Dead ahead. 

With appropriate telemetry, the following is unnecessary 
146:03:03 Ceman: We're at 103, 3.4. 

146:03:06 Parker: Copy that. (Pause) 


A smart system could replace reading the checklist; should 
this be all verbal? 

146:10:11 Ceman: (Looking at checklist page.. . .) Okay, 
and I want to go about (on a heading of) 120. 

Here Bob provides advice, Jack asks for clarification; Bob 
defers to their understanding. If time delay weren't a 
factor, an agent could be getting information off-line from 
Earth or a crew at the Libration Point. 

146:17:08 Parker: Copy that. And just press on the same 
heading you've been carrying there. Gene, and that will get 
you to Camelot. 

146:17:17 Ceman: We want the southwestern edge, huh? 
146:17:20 Schmitt: Do you want to go where Station 5 is. 
Bob (that is, to the planned spot on the southwest rim)? 
146:17:23 Parker: That's my understanding. Jack. So press 
on towards there unless I tell you otherwise. 

146:17:29 Schmitt: Well, but you were talking about 
changing Station 5. 1 think Station 5 is a pretty good spot 
(as is). 

146:17:34 Parker: Roger. And I think that's where we want 
to go. I'm just trying to verify that. You can go in that 
direction, though. I'll get with you if it's not. 

146:17:42 Schmitt: Okay. (Pause) It's probably the most 
concentrated boulder field on Camelot. 

146:17:48 Parker: Okay. You know where it is, and we 
think it's about 092 and 1.6. 

146:17:56 Ceman: 092 and 1.6. You know this country... 
146:18:00 Parker: Roger. But you know where it is, so 
you'll find it when you get there. 

Crew is confused about what they are seeing; Bob 
interrupts 

146:22:37 Ceman: Yeah, because Horatio's got to be on 
our right. Well, wait a minute, doggone it. 

146:22:41 Schmitt: It's not Horatio, is it? 

146:22:43 Ceman: Well, we're at 094, 1.7. 

146:22:46 Parker: Standby. (Pause) 

[They are approaching the west rim of Camelot.] 

146:22:52 Schmitt: No, I think that's Camelot. Horatio 
didn't... 

146:22:53 Ceman: That's too.. .That’s too... 

146:22:54 Schmitt: ...have blocks that far up the rim. 
146:22:55 Ceman: ...Let me... Yeah, let me look in the 
bottom. I'll tell you. I remember. 

_146:22j58 Schmitt; Yeah. 

146:23:00 Parker: That kind of sounds like Camelot to us. 


COMPLICATING ASPECTS OF THE 
WORK INTERACTION 

The crew sometimes mishears or does not listen to each 
other and CapCom. or misunderstands who is speaking 
to whom 

145:28:50 Ceman: Well, I’m going to clean their glasses so 
they know we're (not crazy).. .(To Fendell) Can you wait a 
minute on that pan you’re taking? 



[Gene moves the TV so that he can dust it,] 

145:28:56 Schmitt: I already took it. 

145:28:57 Ceman: No, I mean the television camera. (To 
Fendell) I'll put you back where I had you. 

Lack of immediate response for advice suggests 
communication breakdown 

146:50:03 Ceman: ... When do you want us to leave, Bob? 
(No answer; pause) 

146:50:20 Ceman: Jack, do you read me? 

146:50:21 Schmitt: Yeah. (Pause) Hello, Houston. 
146:50:24 Parker: Hello, 17. Loud and clear. We'd like you 
to leave immediately, if not sooner. 

Dialog is mixed-initiative with interleaved topics 

Here Bob makes a new request, while Cernan and Schmitt 
respond in turn to two different tasks they are doing 
145:54:03 Parker: And what's your frame count, Jack? 
145:54:05 Ceman: Charge number 1. 

145:54:07 Schmitt: Okay, (SEP) power's On, recorder's 
On, the temperature is one- twelve (112). 

145:54:16 Parker: I copy that. 

CapCom loses track of what the crew is doing 
145:54:38 Parker: And, Jack, what's your frame count, 
please? 

145:54:42 Schmitt: Wait, Bob, I can give you that on the 
Rover. 

[TVofn 

145:54:45 Parker: Okay. I thought you were on there. 

CapCom’s remarks were sometimes disruptive and 
unnecessary 

145:55:45 Ceman: That's all right. We got a flag on the 
Rover, and I'm reading 136 on battery number 2. 

145:55:54 Parker: Say again on that one. Gene. 

[Schmitt - "My first inclination in the MOCR would have 
been to turn around and say 'Did anybody get that?, ' rather 
than call right up and distract the crew, '] 

Under time pressure, decisions need to be made about 
competing goals 

145:49:34 Ceman: I got to take a couple of more pictures 
at that contact slope over there. You can't see it from where 
you are. Jack, but I guess we got to leave. Otherwise it 
would be nice to sample that dark stuff up on top. “ ' 
145:49:45 Parker: We need you guys rolling in 7 minutes. 

Mission control made scheduling decisions that were 
non-ODtimal based on opportunities visible to the EVA 
crew 

145:56:21 Parker: Roger. You're moving exactly 37 
seconds early. 

145:56:28 Ceman: Early!? I could have gotten that dark 
mantle on the other side of that crater. That's all it would 
have taken me. 


CapCom’s decisions were based on interactions with 
other people: these conversations are f apparently) not 
documented 

145:32:21 Schmitt: Hey, you want any of this bagged in 
the can. Bob? Canned in the bag.. .or whatever it is? 
145:32:30 Parker: Stand by. They're debating that right 
now. 

The crew works independently and must periodically 
ascertain each other^s status and provide advice to each 
other 

146:34:52 Schmitt: (Turning toward the Rover) How you 
coming, Geno? 

146:34:53 Ceman: Oh, I've got new bags. I've got new 
mags. I've got everything cleaned up and Mark, gravimeter. 
146:35:00 Parker: Copy. Mark that. 

The crew confuses LM location (checklist) with SEP 
transmitter (Rover indicator) 

146:55:56 Schmitt: It must be pretty close. 

146:55:59 Ceman: You bet your life! (Obviously pleased 
with himself) I’m reading 085/1 .4, and that's what my 
checklist said. 

146:56:03 Parker: Roger. (Pause) 

[Again, Gene is misreading his checklist.] 

Photographs may be deficient; quick feedback would 
help 

145:39:26 Ceman: Take your picture. 

[This photo, like the others Jack has taken since finishing 
his pan, is badly overexposed.] 

DISCUSSION 

Anyone looking back over the past few decades must be 
impressed at what we have accomplished in bringing 
computer systems to real-world, complex 
environments — even to imagine assisting astronauts on 
Mars. In large part, our success is enabled by the smaller, 
cheaper, and more reliable and networked personal 
computers of today. That we can imagine 
networking — with components off-the-shelf — ^a half-dozen 
or more supercomputer laptops running on robots, vehicles, 
and backpacks, distributed over many kilometers (Clancey, 
et al., 2003)is truly astounding from the perspective of 
those who stmggled with punch cards and line printers to 
develop AI programs in the 1970s. 

But with this technology bonus, the tables are turned. 
The burden is no longer to just show that the computer can 
do something human-like, such as converse in a dialogue 
or move down a corridor. Today we must confront the 
reality of the environments in which we seek to do 
practical work. The question then becomes, how can 
computers help people? This question is inherently 
empirical, though the answers will be determined by the 
new technologies themselves, developed in the context of 



use, as is generally well known (e.g., see Greenbaum and 
Kyng, 1991), 

This paper was prepared for a symposium, '‘Human 
Interaction with Autonomous Systems in Complex 
Environments.” My chosen title deliberately reverses the 
ordering: Agent interactions with human systems. The 
system is not a technology, but the whole combination of 
people, their tools, and the environment. The symposium 
abstract states: 

Autonomy changes the nature of human tasks and can 
introduce new risks. Mitigating those risks raises 
issues in autonomous systems research such as: 1) 
How to accept task inputs from humans; 2) How to 
adjust the level of autonomy and/or change the 
distribution of roles and responsibilities between 
autonomous systems and humans; 3) How to model 
humans and their tasks and to what level of detail and; 

4) How to facilitate human understanding of the goals, 
tasks and contexts of autonomous systems.,.. 

These issues are real enough, but perhaps especially 
serious if the autonomous system is given, as something 
that must be mitigated. The proposed solutions — accepting 
inputs, adjusting the roles, modeling people, and 
facilitating understanding — fit the traditional view of 
design: From idealized functions (i.e., a superhuman 
teammate) develop technology, adjust the technology to be 
more usable, and then train people to cope with the 
resulting difficulties. 

A human-centered approach (e.g., see Norman, 1998) 
starts instead with the people in their work environment. 
The science of human interaction — as a perceptual-motor, 
cognitive, and social phenomenon — becomes the 
foundation of work system design, including organizations, 
facilities, tools, procedures. This perspective reframes the 
problem, to paraphrase the original abstract: 

Organizations, facilities, tools, and formal procedures 
change the nature of human activity and can introduce 
new opportunities for action. Realizing the advantages 
raises issues in work systems design research such as: 

1) How to determine how people will do their work in 
complex environments that do not yet exist; 2) How 
people will communicate with and learn from each 
other; 3) How people will exchange roles and 
responsibilities with each other over time; 4) How 
tools can facilitate routine tasks, as well as action in 
..dangerous, unexpected, situations.; 5) HowJo .model a 
work system, including people, facilities, geography, 
tools, and procedures and to what level of detail; and 
6) How to facilitate human understanding of the 
operational capabilities and shortcomings of 
autonomous systems .... 

The original perspective of the abstract is not wrong, but it 
is one-sided. It appears to put the burden on fixing the tools 
or fixing the people, rather than grounding the original 
objectives in a better understanding of how people work 
together and how to facilitate their collaboration. 

The symposium abstract suggests “substantial challenges 
in the design of the autonomous systems themselves and in 


the representation and use of the cognitive models 
underpinning human interaction with autonomous 
systems.” This fits the human-centered approach, except 
that again it places the scientific emphasis on studying 
people interacting with tools, rather than people interacting 
with each other. (Consider trying to help carpenters by only 
asking, “How do hammers change the nature of 
carpentry?”) Thus, the models required are not only 
cognitive, but social and perceptual-motor (Clancey, 
2002a). 

In Brahms (Clancey, et al., 1998), we have worked for a 
decade to provide such a modeling framework, in which 
computer systems (whether databases or robots) are 
described and simulated side-by-side with models of 
groups of people and their activities, within a modeled 
geography including buildings, tables, etc. and other tools, 
such as sample bags. Furthermore, we have shown through 
a series of models that it is advantageous to model a full 
day (and indeed a series of days) in order to understand 
how work actions (down to the task level) are affected by 
the context of everyday life (e.g., see Acquisti, et al., 
2002 ). 

Despite my emphasis on empirical studies of people, 
some degree of a priori, imaginative, top-down 
brainstorming is valuable. This suggests roles for 
technology to assist human exploration, such as the 
following (NeXT, 2002): 

1. Robot tracks an EVA crew member while 
carrying tools and a camera. 

2. Robots do site survey and preparation as well as 
post-EVA documentation. 

3. Robots carry tools, which they hand to the EVA 
crew member. Robots can also collect designated 
samples. 

4. Robots physically interact with humans via high- 
level voice commands and gestures. 

5. Robots that are true teammates with humans, 
working on same tasks, responding to natural 
language, gestures and high-level goals and 
recognizing human intentions. 

6. Synergistic relationship between human and 
machine with direct, physical connections and 
prostheses, i.e., super humans augmented with 
machines. 

The list is in many ways consistent with the analysis 
_pres^e.ntedJn this paper, iiov^evex, by Tocusing on xobots 
(the hammers), notice no mention is made of CapCom or 
how to facilitate crew members’ interaction with each 
other. This list, titled “surface assistance metrics,” does not 
attempt to solve known problems, but rather starts with a 
single, ill-defined technolo^ — the idea of a robot. This 
lack of grounding is appropriate for brainstorming, but we 
must be aware such lists will include unrealistic 
ideals — “Robots are true teammates with humans, working 
on the same tasks....” A robot teammate would have 
higher-order consciousness, implying not just a role, but an 
identity and complementary personal projects, just like a 
human collaborator. For brainstorming this is fine, but it 
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has nothing to do with what machines will do on Mars in 
the next twenty years. 

To make practical progress, we need to recognize that 
there is qualitative difference between the first four metrics 
and the last two. To make progress — to relate agents and 
robots to human systems in complex environments — a 
straightforward empirical approach is possible. As 
illustrated in this paper, we can begin with simple, ready- 
at-hand phenomena (including analog settings; Clancey, 
2001, 2002b) of people interacting with each other. The 
focus should be on how to facilitate human-human 
interaction, not how to fix human-machine interaction: 
“Create tools that are designed to make the maximal use of 
human perception and understanding without projecting 
human capacities onto the computer” (Winograd & Flores, 
1986, p. 137). 

For the case of assisting astronauts exploring a planetary 
surface, we have the good fortune of having fully 
documented lunar traverses, transcribed with photographs 
and digitized video. The six Apollo explorations represent 
a gold mine of data. The present analysis is just a first, 
rough cut to show what is available and what kinds of 
discoveries can be made. Using the same format, robot 
developers could document and analyze people working 
together in their domain of choice. Tracking conversations 
has been shown to be a good heuristic for understanding 
the work people do in a complex environment (Kukla, 
1992). One can then find categories of displays and model- 
based tools that will support the work. Some of these tools 
will be agent systems, some of these will be combined with 
sensors, and others (robots) will also include effectors. 

When starting with a complex, perhaps dangerous and 
resource-scarce environment, such as Mars, analog 
experience, such as lunar traverses, may reveal a variety of 
easily defined problems that technology can actually help 
solve. Starting with how the people are interacting with 
each other can provide startling changes in perspective. If 
the astronauts on Apollo are rarely working together, does 
this mean that a robot who is a “true teammate” would be 
off working alone? Or might the entire practice of surface 
exploration be changed by adding a third person? This 
question is more quickly answered by analyzing 
appropriate groups of three people, than by adding a 
rudimentary computer system to a group of two people. 
This argues that requirements analysis should be empirical, 
grounded first and foremost in the study .of human systems. 
By better understanding how people interact with each 
other, we will formulate many practical ideas for what 
computer systems can do and how they should behave. 
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